ATTENTION MECHANISMS

Forecasting Firm Material Events from 8-K Reports (D19-51)

Challenge:	Abstract meaning representation (AMR)-to-text generation is challenging task for natural language processing.
Approach:	They propose a graph-to-sequence model that directly encodes AMR graphs and learns node representations.
Outcome:	The proposed model outperforms the current state-of-the-art neural approach by 1.5 BLEU points on LDC2015E86 and 4.8 BLUE points on the LDC2017T10 and achieves new state- of-the art performance.

Copied to clipboard

Shuang (Sophie) Zhai, Zhu (Drew) Zhang

Challenge:	In this paper, we show deep learning models can be used to forecast firm material event sequences based on the contents of the company’s 8-K Current Reports.
Approach:	They exploit state-of-the-art neural architectures, including sequence-to-sequence architecture and attention mechanisms, to build a deep learning model that can forecast firm material event sequences based on company 8-K Current Reports.
Outcome:	The proposed model can forecast firm material event sequences based on the contents of the firm's 8-K Current Reports.

Efficient Transformer Knowledge Distillation: A Performance Review (2023.emnlp-industry)

Copied to clipboard

Nathan Brown, Ashton Williamson, Tahj Anderson, Logan Lawrence

Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers (2021.acl-srw)

Challenge:	Pretrained transformer language models have been gaining popularity in the field of natural language processing . however, there is no study into the intersection of these two fields .
Approach:	They propose a method to extract knowledge from transformers to produce high-performing efficient attention models with low costs.
Outcome:	The proposed model compression method preserves up to 98.6% of original model performance across short-context tasks and up to 95.8% on long-concept Named Entity Recognition tasks while decreasing inference times by up to 57%.

Copied to clipboard

Jules Samaran, Noa Garcia, Mayu Otani, Chenhui Chu, Yuta Nakashima

PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable (2020.acl-main)

Challenge:	a growing body of research has been focused on what attention heads learn during the pre-training of visual grounded language models.
Approach:	They propose to use visual grounding to supervise attention directly to learn visual ground.
Outcome:	The proposed method improves the performance of a state-of-the-art visual grounded language model on vision-and-language tasks.

Copied to clipboard

Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang

Dodrio: Exploring Transformer Models with Interactive Visualization (2021.acl-demo)

Challenge:	Existing pre-training models for dialogue generation have been proven effective for a wide range of tasks.
Approach:	They propose a dialogue generation pre-training framework that leverages bi-directional context and uni-directional characteristic of language generation.
Outcome:	The proposed framework is superior to existing models on three publicly available datasets.

Copied to clipboard

Zijie J. Wang, Robert Turko, Duen Horng Chau

Query and Extract: Refining Event Extraction as Type-oriented Binary Decoding (2022.findings-acl)

Challenge:	Recent research suggests the key may lie in multi-headed attention mechanism’s ability to learn and represent linguistic information.
Approach:	They present an open-source visualization tool to analyze attention mechanisms in transformer-based models with linguistic knowledge.
Outcome:	Dodrio analyzes attention mechanisms in transformer-based models with linguistic knowledge.

Copied to clipboard

Sijia Wang, Mo Yu, Shiyu Chang, Lichao Sun, Lifu Huang

IFlyLegal: A Chinese Legal System for Consultation, Law Searching, and Document Analysis (D19-3)

Challenge:	Existing approaches to event extraction are limited to a set of pre-defined types.
Approach:	They propose a natural language query framework that uses event types and argument roles to extract candidate triggers and arguments from input text.
Outcome:	The proposed framework outperforms existing methods on zero-shot event extraction.

Copied to clipboard

Ziyue Wang, Baoxin Wang, Xingyi Duan, Dayong Wu, Shijin Wang, Guoping Hu, Ting Liu

Information Retrieval

Challenge:	Legal Tech is a system that performs legal consulting, multi-way law searching, and legal document analysis using deep contextual representations and various attention mechanisms.
Approach:	They propose a Chinese legal system that performs legal consulting, multi-way law searching, and legal document analysis using deep contextual representations and various attention mechanisms.
Outcome:	The proposed system performs legal consulting, multi-way law searching, and legal document analysis by exploiting techniques such as deep contextual representations and various attention mechanisms.

Improving Relation Extraction with Knowledge-attention (D19-1)

Copied to clipboard

Pengfei Li, Kezhi Mao, Xuefeng Yang, Qi Li

Interrogating the Explanatory Power of Attention in Neural Machine Translation (D19-56)

Challenge:	Existing attention mechanisms are data-driven, but most are data driven.
Approach:	They propose a knowledge-attention encoder which integrates prior knowledge from external lexical resources into deep neural networks for relation extraction task.
Outcome:	The proposed system outperforms existing CNN, RNN, and self-attention based models on a large-scale relation extraction dataset.

Copied to clipboard

Pooya Moradi, Nishant Kambhatla, Anoop Sarkar

Machine Translation

Challenge:	Attention models are often used to justify the model’s decision in generating a token but it has not been rigorously established to what extent attention is a reliable source of information in NMT.
Approach:	They propose to use attention models to modify crucial aspects of the trained attention model to produce function and content words in the translation process.
Outcome:	The proposed models preserve function and content words in the translation process compared to state-of-the-art models.

Bidirectional Masked Self-attention and N-gram Span Attention for Constituency Parsing (2023.findings-emnlp)

Copied to clipboard

Soohyeong Kim, Whanhee Cho, Minji Kim, Yong Choi

Learning Dynamic Context Augmentation for Global Entity Linking (D19-1)

Challenge:	Existing attention mechanisms for constituency parsing lack directional information needed to form sentence spans.
Approach:	They propose a bidirectional masked and N-gram span Attention model which captures the explicit dependencies between each word and enhances the representation of the output span vectors.
Outcome:	The proposed model achieves state-of-the-art performance on the Penn Treebank and Chinese Penn TreeBank datasets with F1 scores of 96.47 and 94.15 respectively.

Copied to clipboard

Xiyuan Yang, Xiaotao Gu, Sheng Lin, Siliang Tang, Yueting Zhuang, Fei Wu, Zhigang Chen, Guoping Hu, Xiang Ren

Knowledge Representation

Challenge:	Existing collective entity linking methods are expensive and often lack local context information.
Approach:	They propose a dynamic context-augmented inference model that can be used to make collective inference.
Outcome:	The proposed model can cope with different local EL models with different learning settings, base models, decision orders and attention mechanisms.

Multiformer: A Head-Configurable Transformer-Based Model for Direct Speech Translation (2022.naacl-srw)

Copied to clipboard

Gerard Sant, Gerard I. Gállego, Belen Alastruey, Marta Ruiz Costa-jussà

Location Attention for Extrapolation to Longer Sequences (2020.acl-main)

Challenge:	Existing approaches to address speech tasks with a self-attention mechanism are expensive and lead to information loss.
Approach:	They propose a Transformer-based model which uses different attention mechanisms on each head to bias the self-attention towards the extraction of more diverse token interactions.
Outcome:	The proposed model outperforms baseline models by 0.7 BLEU in the speech task.

Copied to clipboard

Yann Dubois, Gautier Dagan, Dieuwke Hupkes, Elia Bruni

Challenge:	Neural networks are surprisingly good at interpolating, but they are often unable to extrapolate patterns beyond the seen data.
Approach:	They propose to use a special type of extrapolation for natural language processing to generalize to sequences that are longer than the training ones.
Outcome:	The proposed model is more likely to extrapolate than models with common attention mechanisms.

Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification (2021.acl-long)

Copied to clipboard

George Chrysostomou, Nikolaos Aletras

Explainability & Interpretability in NLP

Challenge:	Recent studies show that attention cannot be considered as a faithful explanation across encoders and tasks.
Approach:	They propose a new family of Task-Scaling mechanisms that scale attention weights across tasks and two attention mechanisms.
Outcome:	The proposed models improve explanation faithfulness across two attention mechanisms, five encoders and five text classification datasets without sacrificing predictive performance.

EdgeInfinite: A Memory-Efficient Infinite-Context Transformer for Edge Devices (2025.acl-industry)

Copied to clipboard

Jiyu Chen, Shuang Peng, Daxiong Luo, Fan Yang, Renshou Wu, Fangyuan Li, Xiaoxin Chen

Context-Aware Neural Model for Temporal Information Extraction (P18-1)

Challenge:	Existing KV cache optimizations struggle with irreversible token eviction in long-output tasks . alternative sequence modeling architectures prove costly to adopt within established Transformer infrastructures.
Approach:	They propose a memory-efficient solution for infinite contexts that integrates compressed memory into Transformer-based LLMs through a trainable memory-gating module.
Outcome:	The proposed solution achieves comparable performance to baseline Transformer-based LLMs while optimizing memory consumption and time to first token.

Copied to clipboard

Yuanliang Meng, Anna Rumshisky

Exploring Attention Attractors in Large Language Models (2026.acl-long)

Challenge:	Existing temporal information extraction systems rely on statistical learning with feature-engineered task-specific models.
Approach:	They propose a context-aware neural network model for temporal information extraction using a global context layer.
Outcome:	The proposed model outperforms existing models in terms of performance and performance . it is the first model to use NTM-like architecture to process the information from global context in discourse-scale natural text processing.

Copied to clipboard

Ziheng Wang, Zihao Yue, Wenxuan Wang, Qin Jin

MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment (2026.acl-industry)

Challenge:	Existing studies have suggested that attention attractors function as "summary tokens" while others speculate that tokens with weaker semantics attract high attention, they act as attention sinks that offload excessive attention.
Approach:	They examine attention attractors, tokens that draw significantly high attention, in large language models.
Outcome:	The proposed models are able to capture long-range dependencies within a given context.

Copied to clipboard

Hanxian Huang, Igor Fedorov, Andrey Gromov, Bernard Beckerman, Naveen Suda, David Eriksson, Maximilian Balandat, Rylan Conway, Patrick Huber, Chinnadhurai Sankar, Ayushi Dalmia, Zechun Liu, Lemeng Wu, Tarek Elgamal, Adithya Sagar, Vikas Chandra, Raghuraman Krishnamoorthi

Progressive Self-Supervised Attention Learning for Aspect-Level Sentiment Analysis (P19-1)

Challenge:	MobileLLM-Flash is a family of foundation models for efficient on-device use with strong capabilities.
Approach:	They propose a method for designing on-device large language models under mobile latency constraints using hardware-in-the-loop architecture search.
Outcome:	The proposed model is amenable to industry-scale deployment and is compatible with mobile runtimes like Executorch.

Copied to clipboard

Jialong Tang, Ziyao Lu, Jinsong Su, Yubin Ge, Linfeng Song, Le Sun, Jiebo Luo

Truth-Conditional Captions for Time Series Data (2021.emnlp-main)

Challenge:	Experimental results show that our proposed approach yields better attention mechanisms . dominant ASC models are mostly discriminative classifiers based on manual feature engineering .
Approach:	They propose a self-supervised approach to aspect-level sentiment classification that mines useful attention supervision information from a training corpus to refine attention mechanisms.
Outcome:	The proposed approach yields better attention mechanisms on multiple datasets.

Copied to clipboard

Harsh Jhamtani, Taylor Berg-Kirkpatrick

Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models (2025.acl-long)

Challenge:	Existing models with attention mechanisms can generate fluent descriptions of salient patterns in time series, but they often generate factually incorrect descriptions.
Approach:	They propose a model which first runs small learned programs on the input time series, then identifies the programs/patterns which hold true for the given input, and finally conditions on only the chosen valid program to generate the output text description.
Outcome:	The proposed model extracts high-level patterns from the data and generates high precision captions even though it is built on a small space of modules.

Copied to clipboard

Junfeng Tian, Da Zheng, Yang Chen, Rui Wang, Colin Zhang, Debing Zhang

Surprisingly Easy Hard-Attention for Sequence to Sequence Learning (D18-1)

Challenge:	Large language models (LLMs) have prioritized expanding the context window from which they can incorporate more information.
Approach:	They propose a data augmentation strategy to enable large language models to gain long-context capabilities without the need to modify existing data mixture.
Outcome:	The proposed model outperforms existing models on 20 billion tokens and achieves 75% and 84.5% accuracy on RULER at 128K context length.

Copied to clipboard

Shiv Shankar, Siddhant Garg, Sunita Sarawagi

Differentiable Subset Pruning of Transformer Heads (2021.tacl-1)

Challenge:	Existing attention mechanisms are hard and hard, but they are more accurate when trained.
Approach:	They propose to use a beam approximation of the joint distribution between attention and output to train sequence to sequence learning.
Outcome:	The proposed method is compared to existing attention mechanisms on five translation tasks and shows consistent gains on the same tasks.

Copied to clipboard

Jiaoda Li, Ryan Cotterell, Mrinmaya Sachan

Relation Logical Reasoning and Relation-aware Entity Encoding for Temporal Knowledge Graph Reasoning (2025.coling-main)

Challenge:	Recent work shows that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model.
Approach:	They propose a method that prunes a Transformer's multi-head attention mechanism away without significantly harming its performance.
Outcome:	The proposed method improves on natural language inference and machine translation tasks while offering precise control of sparsity level.

Copied to clipboard

Longzhou Liu, Chenglong Xiao, Shanshan Wang, Tingwen Liu

Multimodal and Multi-view Models for Emotion Recognition (P19-1)

Challenge:	Current knowledge graph models focus on embedding entities and relations, overlooking the broader structure of the entire knowledge graph.
Approach:	They propose a Temporal Knowledge Graph Reasoning model that embeds relation embeddings into the TKG.
Outcome:	The proposed model outperforms state-of-the-art models on five public datasets . it uses relation-aware attention mechanisms to learn relation embeddings based on query relations .

Copied to clipboard

Gustavo Aguilar, Viktor Rozgic, Weiran Wang, Chao Wang

Attention Calibration for Transformer in Neural Machine Translation (2021.acl-long)

Challenge:	combining lexical and acoustic information results in more robust and accurate models . combining both modalities may be a bottleneck in a deployment pipeline due to computational complexity or privacy constraints .
Approach:	They propose to combine acoustic and lexical information to provide a deployable acustic model . they use multimodal models and two attention mechanisms to assess the benefits of lexicals .
Outcome:	The proposed model outperforms the state-of-the-art on the USC-IEMOCAP dataset . it significantly surpasses models that have been exclusively trained with acoustic features .

Copied to clipboard

Yu Lu, Jiali Zeng, Jiajun Zhang, Shuangzhi Wu, Mu Li

An Evaluation of Neural Machine Translation Models on Historical Spelling Normalization (C18-1)

Challenge:	Attention mechanisms have been ubiquitous in neural machine translation (NMT) however, many studies doubt whether highlyattended inputs have a large impact on the model outputs.
Approach:	They propose to introduce a mask perturbation model that automatically evaluates each input’s contribution to the model outputs.
Outcome:	The proposed model is more uniform at lower layers while more concentrated on the specific inputs at higher layers.

Copied to clipboard

Gongbo Tang, Fabienne Cap, Eva Pettersson, Joakim Nivre

Machine Translation

Challenge:	In this paper, we apply different NMT models to the problem of historical spelling normalization for five languages . we find that NMT model is much better than SMT in terms of character error rate .
Approach:	They propose to use NMT models to solve the problem of historical spelling normalization in five languages.
Outcome:	The proposed method improves historical spelling normalization for five languages.

Unsupervised Recurrent Neural Network Grammars (N19-1)

Copied to clipboard

Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, Gábor Melis

A Multi-sentiment-resource Enhanced Attention Network for Sentiment Classification (P18-2)

Challenge:	RNNGs model syntax and structure by incrementally generating a syntax tree and sentence in a top-down, left-to-right order.
Approach:	They explore unsupervised learning of recurrent neural network grammars for language modeling and grammar induction.
Outcome:	The proposed model outperforms standard sequential language models and improves parsing performance.

Copied to clipboard

Zeyang Lei, Yujiu Yang, Min Yang, Yi Liu

Attention-Based Capsule Networks with Dynamic Routing for Relation Extraction (D18-1)

Challenge:	Existing sentiment classification approaches do not fully exploit sentiment linguistic knowledge.
Approach:	They propose a Multi-sentiment-resource Enhanced Attention Network to integrate sentiment linguistic knowledge into the deep neural network via attention mechanisms.
Outcome:	The proposed network captures sentiments from different representation sub-spaces, and is superior to strong competitors.

Copied to clipboard

Ningyu Zhang, Shumin Deng, Zhanling Sun, Xi Chen, Wei Zhang, Huajun Chen

Modeling Recurrence for Transformer (N19-1)

Challenge:	Existing neural networks focus on instance representation, and subsampling fails to retain precise spatial relationships between higher-level parts.
Approach:	They propose a neural approach based on capsule networks with attention mechanisms to extract relational information from a capsule.
Outcome:	The proposed method improves the precision of the predicted relations with different benchmarks.

Copied to clipboard

Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, Zhaopeng Tu

Using a Penalty-based Loss Re-estimation Method to Improve Implicit Discourse Relation Classification (2020.coling-main)

Challenge:	Existing studies show that the lack of recurrence modeling hinders the development of a translation model.
Approach:	They propose to model recurrence for Transformer with an additional recurrent encoder.
Outcome:	The proposed model outperforms the deep model on EnglishGerman and ChineseEnglish translation tasks.

Copied to clipboard

Xiao Li, Yu Hong, Huibin Ruan, Zhen Huang

Text Classification

Challenge:	inessential words are unintentionally misjudged as attention-worthy words and assigned heavier attention weights than should be.
Approach:	They propose a penalty-based method to regulate the attention learning process by integrating penalty coefficients into the computation of loss by means of overstability of attention weight distributions.
Outcome:	The proposed method improves on the Penn Discourse TreeBank corpus and is competitive compared to the state-of-the-art methods.

ASEM: Enhancing Empathy in Chatbot through Attention-based Sentiment and Emotion Modeling (2024.lrec-main)

Copied to clipboard

Omama Hamad, Khaled Shaban, Ali Hamdi

Enhancing Long-range Dependency with State Space Model and Kolmogorov-Arnold Networks for Aspect-based Sentiment Analysis (2025.coling-main)

Challenge:	Existing models lack feature representations that capture the deep semantics of language and sensitivity to minor input variations, resulting in significant changes in the generated text.
Approach:	They propose an end-to-end model architecture called ASEM that performs emotion analysis on top of sentiment analysis for open-domain chatbots.
Outcome:	The proposed model outperforms existing models for generating empathetic embeddings, providing e-mpathetic and diverse responses.

Copied to clipboard

Adamu Lawan, Juhua Pu, Haruna Yunusa, Aliyu Umar, Muhammad Lawan

Octa: Omissions and Conflicts in Target-Aspect Sentiment Analysis (2020.findings-emnlp)

Challenge:	Aspect-based Sentiment Analysis (ABSA) evaluates sentiments toward specific aspects of entities within the text.
Approach:	They propose a method to enhance long-range dependencies between aspect and opinion words in ABSA by combining attention mechanisms with a syntax-based Graph Convolutional Network and a Mamba-Transformer module.
Outcome:	The proposed model outperforms state-of-the-art models on three benchmark datasets.

Copied to clipboard

Zhe Zhang, Chung-Wei Hang, Munindar Singh

Generative Knowledge Selection for Knowledge-Grounded Dialogues (2023.findings-eacl)

Challenge:	Existing sentiment analysis models treat aspects and targets separately, causing conflicting sentiments.
Approach:	They propose an approach that jointly considers aspects and targets when inferring sentiments.
Outcome:	The proposed approach outperforms leading models by 1.6% to 4.3% on benchmark datasets . it uses selective attention mechanisms for selective attention between targets and context words .

Copied to clipboard

Weiwei Sun, Pengjie Ren, Zhaochun Ren

RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER (2020.coling-main)

Challenge:	Knowledge selection is the key in knowledge-grounded dialogues (KGD), which aims to select an appropriate knowledge snippet to be used in the utterance based on dialogue history.
Approach:	They propose a generative approach for knowledge selection called GenKS that learns to select snippets by generating their identifiers with a sequence-to-sequence model.
Outcome:	The proposed approach captures intra-knowledge interaction inherently through attention mechanisms while generating their identifiers with a sequence-to-sequence model.

Copied to clipboard

Lin Sun, Jiquan Wang, Yindu Su, Fangsheng Weng, Yuxuan Sun, Zengwei Zheng, Yuanyi Chen

Challenging Reading Comprehension on Daily Conversation: Passage Completion on Multiparty Dialog (N18-1)

Challenge:	Named entity recognition (MNER) for tweets is a key task of many applications.
Approach:	They propose a pre-trained multimodal named entity recognition model based on Relationship Inference and Visual Attention (RIVA) for tweets.
Outcome:	The proposed model improves on the multimodal named entity recognition (MNER) task on tweets with the aid of visual clues.

Copied to clipboard

Kaixin Ma, Tomasz Jurczyk, Jinho D. Choi

MIRTT: Learning Multimodal Interaction Representations from Trilinear Transformers for Visual Question Answering (2021.findings-emnlp)

Challenge:	Existing approaches to reading comprehension on multiparty dialogs have focused on children's stories or newswire.
Approach:	They propose a new corpus and a robust deep learning architecture for a task in reading comprehension on multiparty dialog.
Outcome:	The proposed model outperforms the state-of-the-art model on a different genre using bidirectional LSTM, showing a 13.0+% improvement for longer dialogs.

Copied to clipboard

Junjie Wang, Yatai Ji, Jiaqi Sun, Yujiu Yang, Tetsuya Sakai

To Attend or not to Attend: A Case Study on Syntactic Structures for Semantic Relatedness (P18-1)

Challenge:	Existing bilinear methods focus on inter-modality information between images and questions . existing models focus on the interaction between images, questions, and images .
Approach:	They propose a trilinear interaction framework that incorporates attention mechanisms for capturing inter-modality and intra-modal relationships.
Outcome:	The proposed model outperforms bilinear models on the Visual7W Telling task and VQA-1.0 Multiple Choice task and outperformed baselines on the VQA, TDIUC and GQA datasets.

Copied to clipboard

Amulya Gupta, Zhu Zhang

Explicit Utilization of General Knowledge in Machine Reading Comprehension (P19-1)

Challenge:	Recent success of Recurrent Neural Networks (RNNs) in Machine Translation (MT) has prompted attention mechanisms to be used in machine translation.
Approach:	They propose a tree-structured attention model on Tree Long Short-Term Memory Networks . they also experiment with three LSTM variants: bidirectional-LSTMs, Constituency Tree-LSTS, and Dependency Tree LSTS.
Outcome:	The proposed model is based on tree-LSTMs, constituency trees, and dependencies trees.

Copied to clipboard

Chao Wang, Hui Jiang

Reasoning

Challenge:	Existing MRC models are unable to integrate general knowledge with human knowledge.
Approach:	They propose a data enrichment method which uses WordNet to extract inter-word semantic connections as general knowledge from each given passage-question pair.
Outcome:	The proposed model outperforms state-of-the-art models and is robust to noise.

A Neural Transition-based Joint Model for Disease Named Entity Recognition and Normalization (2021.acl-long)

Copied to clipboard

Zongcheng Ji, Tian Xia, Mei Han, Jing Xiao

Why Attention is Not Explanation: Surgical Intervention and Causal Reasoning about Neural Models (2020.lrec-1)

Challenge:	Existing models for disease recognition and normalization ignore text surface form of each candidate concept, causing boundary inconsistency.
Approach:	They propose a neural transition-based joint model to normalize disease entities from biomedical text.
Outcome:	The proposed model improves on two publicly available datasets.

Copied to clipboard

Christopher Grimsley, Elijah Mayfield, Julia R.S. Bursten

Enhancing Out-of-Vocabulary Estimation with Subword Attention (2023.findings-acl)

Challenge:	a recent study finds brittleness in explanations obtained through attention mechanisms . a philosophy of science theory allows robust yet non-causal reasoning in explanation .
Approach:	They propose to use philosophy of science to examine the state-of-the-art in explanation for NLP models . they argue that it is impossible to explain attention-based learning by attention mechanisms .
Outcome:	The proposed model selection criteria are based on philosophy of science theories . the proposed model is based upon a model that is more explainable than a classical model .

Copied to clipboard

Raj Patel, Carlotta Domeniconi

Negative Focus Detection via Contextual Attention Mechanism (D19-1)

Challenge:	Existing methods to learn OOV word representations use advanced architectures like attention on the context of the word, but they tend to use simple structures like ngram addition or character based convolutional neural networks (CNN)
Approach:	They propose a transformer-based OOV estimation model that uses attention mechanisms on both the context and the subwords to learn OOV representations.
Outcome:	The proposed model outperforms current state-of-the-art models on OOV representations based on attention mechanisms on the context and subwords .

Copied to clipboard

Longxiang Shen, Bowei Zou, Yu Hong, Guodong Zhou, Qiaoming Zhu, AiTi Aw

Challenge:	Negation is a universal but complicated linguistic phenomenon that reverses the polarity of a statement or its property into opposite.
Approach:	They propose a framework which consists of a Bidirectional Long Short-Term Memory neural network and a Conditional Random Fields layer to capture contextual information.
Outcome:	The proposed framework improves on the SEM’12 shared task corpus, yielding an absolute improvement of 2.11% over the state-of-the-art.

Aspect Based Sentiment Analysis with Gated Convolutional Networks (P18-1)

Copied to clipboard

Wei Xue, Tao Li

Encoding and Fusing Semantic Connection and Linguistic Evidence for Implicit Discourse Relation Recognition (2022.findings-acl)

Challenge:	Aspect-based sentiment analysis can provide more detailed information than general sentiment analysis.
Approach:	They propose a model based on convolutional neural networks and gating mechanisms which can selectively output the sentiment features according to the given aspect or entity.
Outcome:	The proposed model can selectively output sentiment features according to the given aspect or entity.

Copied to clipboard

Wei Xiang, Bang Wang, Lu Dai, Yijun Mo

Challenge:	Existing studies use one attention mechanism to improve contextual semantic representation learning for implicit discourse relation recognition (IDRR).
Approach:	They propose a Multi-Attentive Neural Fusion model to fuse linguistic evidence and semantic connection for IDRR by using a Dual Attention Network and an Offset Matrix Network.
Outcome:	The proposed model achieves state-of-the-art on the PDTB 3.0 corpus.

Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings (2020.emnlp-main)

Copied to clipboard

Yue Wang, Jing Li, Michael Lyu, Irwin King

VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020.emnlp-main)

Challenge:	Existing studies focus on text modeling, ignoring the rich features embedded in the matching images.
Approach:	They propose a novel multi-modal multi-head attention model to capture cross-media interactions and image wordings to bridge the two modalities.
Outcome:	The proposed model outperforms the current state of the art based on text modeling and image matching .

Copied to clipboard

Yue Wang, Shafiq Joty, Michael Lyu, Irwin King, Caiming Xiong, Steven C.H. Hoi

Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models (2025.findings-acl)

Challenge:	Prior work focused on attention mechanisms to model complex interactions in visual dialog . a new framework for visual dialog is based on pretrained BERT language models .
Approach:	They propose a framework for a vision-dialog Transformer that leverages pretrained BERT language models for Visual Dialog tasks.
Outcome:	The proposed framework achieves the top position on the visual dialog leaderboard without pretraining on external vision-language data.

Copied to clipboard

Huangyw Huangyw, Yong Zhang, Ning Cheng, Zhitao Li, Shaojun Wang, Jing Xiao

Interactive Key-Value Memory-augmented Attention for Image Paragraph Captioning (2020.coling-main)

Challenge:	Existing methods, such as a n-terminal coding, do not provide accurate data for large language models.
Approach:	They propose a lightweight framework that leverages attention distributions and uncertainty signals in a single-pass decoding.
Outcome:	Experiments on open-book QA datasets show that DAGCD improves faithfulness and robustness while preserving computational efficiency.

Copied to clipboard

Chunpu Xu, Yu Li, Chengming Li, Xiang Ao, Min Yang, Jinwen Tian

Bridge the Gap: High-level Semantic Planning for Image Captioning (2020.coling-main)

Challenge:	Existing approaches to image paragraph captioning ignore the past alignment information, resulting in repetitive captioning and incomplete captioning.
Approach:	They propose an Interactive key-value Memory-augmented Attention model for image paragraph captioning to keep track of attention history along with update-chain of decoder state.
Outcome:	Extensive experiments on a benchmark dataset demonstrate the effectiveness of the proposed model.

Copied to clipboard

Chenxi Yuan, Yang Bai, Chun Yuan

Interactively-Propagative Attention Learning for Implicit Discourse Relation Recognition (2020.coling-main)

Challenge:	Recent image captioning models have improved the multi-modal interaction, such as attention mechanisms.
Approach:	They propose a high-level semantic planning mechanism that integrates a semantic reconstruction and an explicit order planning mechanism to bridge the gap between visual and language domains.
Outcome:	The proposed model outperforms previous methods and achieves the state-of-the-art performance on MS COCO.

Copied to clipboard

Huibin Ruan, Yu Hong, Yang Xu, Zhen Huang, Guodong Zhou, Min Zhang

Challenge:	Existing models for discourse relation recognition use self-attention and interactive-attention mechanisms.
Approach:	They develop a propagative attention learning model using a cross-coupled two-channel network.
Outcome:	The proposed model improves on the baseline models on a Penn Discourse Treebank.

Interpretable Neural Predictions with Differentiable Binary Variables (P19-1)

Copied to clipboard

Jasmijn Bastings, Wilker Aziz, Ivan Titov

Explainability & Interpretability in NLP

Challenge:	Neural networks are bringing incredible performance gains on text classification tasks, but they also require interpretability.
Approach:	They propose a latent model that selects a rationale and a classifier that learns from the words in the rationale alone.
Outcome:	The proposed model can predict expected value of penalties without REINFORCE and can be directly optimised towards a pre-specified text selection rate.

Dynamic Multistep Reasoning based on Video Scene Graph for Video Question Answering (2022.naacl-main)

Copied to clipboard

Jianguo Mao, Wenbin Jiang, Xiangdong Wang, Zhifan Feng, Yajuan Lyu, Hong Liu, Yong Zhu

DLM: A Decoupled Learning Model for Long-tailed Polyphone Disambiguation in Mandarin (2024.naacl-long)

Challenge:	Existing video QA models lack the capacity for deep video understanding and flexible multistep reasoning.
Approach:	They propose a video question answering model which performs dynamic multistep reasoning between questions and videos.
Outcome:	The proposed model improves on three widely used video QA datasets and displays better interpretability by backtracing along with the attention mechanisms to the video scene graphs.

Copied to clipboard

Beibei Gao, Yangsen Zhang, Ga Xiang, Yushan Jiang

Low-Resource NLP

Challenge:	Grapheme-to-phoneme conversion datasets suffer from the long-tail problem . context learning for polyphonic characters often stems from a single dimension .
Approach:	They propose a model for long-tailed polyphone disambiguation in Mandarin that decouples representation and classification learnings.
Outcome:	The proposed model can decouple representation and classification learnings . it achieves transition learning of context from local to global .

A Richer-but-Smarter Shortest Dependency Path with Attentive Augmentation for Relation Extraction (N19-1)

Copied to clipboard

Duy-Cat Can, Hoang-Quynh Le, Quang-Thuy Ha, Nigel Collier

Bidirectional Attentive Memory Networks for Question Answering over Knowledge Bases (N19-1)

Challenge:	Existing approaches to extract relationship between entities in sentences suffer from missing or redundant information.
Approach:	They propose a deep neural model that combines the advantages of the two approaches to extract the relationship between two entities in a sentence.
Outcome:	The proposed model outperforms baseline models on the SemEval-2010 dataset.

Copied to clipboard

Yu Chen, Lingfei Wu, Mohammed J. Zaki

Question Answering

Challenge:	Existing methods for knowledge base question answering ignore subtle inter-relationships between the question and the KB.
Approach:	They propose to model the two-way flow of interactions between questions and KBs using a bidirectional attentive memory network.
Outcome:	The proposed method outperforms existing methods on the WebQuestions benchmark and offers better interpretability compared to baselines.

Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge (2024.findings-acl)

Copied to clipboard

Daniel Mela, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas

A Review of Incorporating Psychological Theories in LLMs (2026.eacl-long)

Challenge:	Recent studies have explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks.
Approach:	They propose a method that allows users to edit factual associations without catastrophic forgetting.
Outcome:	The proposed method achieves 10% increase in magnitude metrics while requiring minimal parameter modifications.

Copied to clipboard

Zizhou Liu, Ziwei Gong, Lin Ai, Zheng Hui, Run Chen, Colin Wayne Leach, Michelle R. Greene, Julia Hirschberg

Linguistics & Cognitive NLP

Challenge:	a holistic review systematically integrating psychology across the LLM lifecycle remains missing.
Approach:	They examine how psychological theories can inform stages of LLM development . they highlight current trends and gaps in how psychological theory is applied .
Outcome:	The authors highlight current trends and gaps in how psychological theories are applied . they argue that psychological insights have shaped pivotal NLP breakthroughs .

On Biasing Transformer Attention Towards Monotonicity (2021.naacl-main)

Copied to clipboard

Annette Rios, Chantal Amrhein, Noëmi Aepli, Rico Sennrich

Towards Transparent and Explainable Attention Models (2020.acl-main)

Challenge:	Existing work has focused on learning monotonic attention behavior via specialized attention functions or pretraining.
Approach:	They introduce a monotonicity loss function compatible with standard attention mechanisms and test it on sequence-to-sequence tasks.
Outcome:	The proposed monotonicity loss function can achieve largely monotonic behavior on grapheme-to-phoneme conversion, morphological inflection, transliteration, and dialect normalization tasks.

Copied to clipboard

Akash Kumar Mohankumar, Preksha Nema, Sharan Narasimhan, Mitesh M. Khapra, Balaji Vasan Srinivasan, Balaraman Ravindran

Explainability & Interpretability in NLP

Challenge:	Recent studies on interpretability of attention distributions have led to notions of faithful and plausible explanations for a model’s predictions.
Approach:	They propose to modify LSTM cells to ensure that the hidden representations learned at different time steps are diverse.
Outcome:	The proposed model can provide a faithful explanation if a higher attention weight implies a greater impact on the model’s prediction.

Self-Adjust Softmax (2025.emnlp-main)

Copied to clipboard

Chuanyang Zheng, Yihang Gao, Guoxuan Chen, Han Shi, Jing Xiong, Xiaozhe Ren, Chao Huang, Zhenguo Li, Yu Li

Sentiment and Emotion help Sarcasm? A Multi-task Learning Framework for Multi-Modal Sarcasm, Sentiment and Emotion Analysis (2020.acl-main)

Challenge:	Usually, tokens with larger attention scores are important for the final prediction.
Approach:	They propose to modify softmax(z) to z softmax and its normalized variant to improve the Transformer attention mechanism by making minor adjustments to the softmax function.
Outcome:	The proposed model provides enhanced gradient properties compared to the vanilla softmax function.

Copied to clipboard

Dushyant Singh Chauhan, Dhanush S R, Asif Ekbal, Pushpak Bhattacharyya

From Text Segmentation to Enhanced Representation Learning: A Novel Approach to Multi-Label Classification for Long Texts (2024.findings-emnlp)

Challenge:	Existing systems for sarcasm detection are limited by the use of sarcasm . sarasm is often used to convey thinly veiled disapproval humorously.
Approach:	They propose a multi-task deep learning framework to solve sarcasm problems simultaneously . they manually annotate a sarcsm dataset with sentiment and emotion classes .
Outcome:	The proposed framework is able to solve sarcasm, sentiment and emotion problems in a multi-modal conversational scenario.

Copied to clipboard

Wang Zhang, Xin Wang, Qian Wang, Tao Deng, Xiaoru Wu

Rethinking Complex Neural Network Architectures for Document Classification (N19-1)

Challenge:	Existing models rely on pre-trained language models, which have a maximum input sequence length of 512 tokens, and therefore have 'input length limitation'.
Approach:	They propose a text segmentation algorithm which guarantees to produce the optimal segmentation to address the issue of input length limitation caused by PLMs.
Outcome:	The proposed method improves both text and label representations on MLTC datasets, unraveling the intricate correlations between texts and labels.

Copied to clipboard

Ashutosh Adhikari, Achyudh Ram, Raphael Tang, Jimmy Lin

Information Retrieval

Challenge:	Neural network models for many NLP tasks have grown increasingly complex in recent years . authors of recent papers question the necessity of such architectures and find them quite effective .
Approach:	They propose to use regularization techniques borrowed from language modeling to improve model accuracy . they find that a simple biLSTM architecture with appropriate regularization yields competitive results .
Outcome:	a simple biLSTM model outperforms the state-of-the-art on four benchmark datasets . authors say that improvements are not real, but are attributed to mundane reasons .

Human Attention Maps for Text Classification: Do Humans and Neural Networks Focus on the Same Words? (2020.acl-main)

Copied to clipboard

Cansu Sen, Thomas Hartvigsen, Biao Yin, Xiangnan Kong, Elke Rundensteiner

Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction (2021.naacl-main)

Challenge:	Attention-based models have been claimed to add interpretability, but little is known about the actual relationships between machine and human attention.
Approach:	They conduct the first quantitative assessment of human versus computational attention mechanisms for the text classification task.
Outcome:	The proposed models are compared against machine attention maps on a publicly available YELP dataset.

Copied to clipboard

Zhenghao Liu, Xiaoyuan Yi, Maosong Sun, Liner Yang, Tat-Seng Chua

Syntactic Text Processing

Challenge:	Existing GEC models produce spurious corrections or fail to detect lots of errors.
Approach:	They propose a neural network for GEC quality estimation with multiple hypotheses . VERNet establishes interactions among hypothese based on reasoning graph .
Outcome:	The proposed model achieves state-of-the-art grammatical error detection performance and best quality estimation results on four GEC datasets.

Attention Understands Semantic Relations (2022.lrec-1)

Copied to clipboard

Anastasia Chizhikova, Sanzhar Murzakhmetov, Oleg Serikov, Tatiana Shavrina, Mikhail Burtsev

Learning to Deceive with Attention-Based Explanations (2020.acl-main)

Challenge:	Present-day monopoly of foundation language models in most tasks forces researchers and practitioners to rely on popular large models without genuinely understanding the models' behaviour.
Approach:	They propose a probing pipeline to study the representedness of semantic relations in transformer language models and propose 'attention mechanisms' that focus on syntactic relational information and semantic one.
Outcome:	The proposed pipeline shows that attention scores are expressive as output activations on this task, despite their lesser ability to represent surface cues.

Copied to clipboard

Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, Zachary C. Lipton

DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination (2024.emnlp-main)

Challenge:	Attention mechanisms are ubiquitous components in neural network architectures and are often claimed to confer interpretability.
Approach:	They propose a method for training models to produce deceptive attention masks by combining weights assigned to designated impermissible tokens with a weighted sum.
Outcome:	The proposed method reduces the weight assigned to designated impermissible tokens while still using them across multiple models and tasks.

Copied to clipboard

Xuan Gong, Tianshi Ming, Xinpeng Wang, Zhihua Wei

Language-Guided Temporal Token Pruning for Efficient VideoLLM Processing (2025.emnlp-main)

Challenge:	Despite the success of Large Vision-Language Models, they suffer from hallucination.
Approach:	They propose a training-free strategy that "Dive into" the attention of LVLMs to "Reduce" object hallucination by using classification tokens of ViT.
Outcome:	The proposed method reduces the impact of outlier tokens on LVLMs . the proposed method is based on LLaVA-1.5, LLvaVA-NeXT and InstructBLIP .

Copied to clipboard

Yogesh Kumar

Visual Data in NLP

Challenge:	Current models struggle with long-form videos due to the quadratic complexity of attention mechanisms.
Approach:	They propose a model-agnostic framework that leverages temporal cues from queries to prune video tokens.
Outcome:	The proposed framework reduces computation by 65% while preserving 97-99% of original performance.

Extracting Shopping Interest-Related Product Types from the Web (2023.findings-acl)

Copied to clipboard

Yinghao Li, Colin Lockard, Prashant Shiralkar, Chao Zhang

If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation? (2025.acl-long)

Challenge:	Existing e-commerce products are limited in their ability to assist customers in interest-oriented shopping.
Approach:	They propose to extract PTs from Web pages containing hand-crafted PT recommendations for SIs . they propose to use tree-transformer encoders for node classification to improve inter-node dependency modeling .
Outcome:	The proposed model outperforms the best baseline model by 2.37 F1 points on a WebPT dataset.

Copied to clipboard

Ryo Yoshida, Shinnosuke Isono, Kohei Kajikawa, Taiga Someya, Yushi Sugimoto, Yohei Oseki

Multi-Label Few-Shot Learning for Aspect Category Detection (2021.acl-long)

Challenge:	Recent work in computational psycholinguistics has revealed intriguing parallels between attention mechanisms and human memory retrieval, focusing primarily on vanilla Transformers that operate on token-level representations.
Approach:	They propose that the attention mechanism of Transformer Grammar (TG) can serve as a cognitive model of human memory retrieval using Normalized Attention Entropy (NAE) they propose that TG's attention can implement a human memory-retrieval theory known as cue-based retrieval .
Outcome:	The attention mechanism of Transformer Grammar (TG) achieves superior predictive power for self-paced reading times compared to vanilla Transformer’s, with further analyses revealing independent contributions from both models.

Copied to clipboard

Mengting Hu, Shiwan Zhao, Honglei Guo, Chao Xue, Hang Gao, Tiegang Gao, Renhong Cheng, Zhong Su

CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels (2026.acl-long)

Challenge:	Existing few-shot learning methods focus on single-label predictions, which can not work well for ACD since a sentence may contain multiple aspect categories.
Approach:	They propose a few-shot learning method that uses the prototypical network to learn aspects from a set of aspects.
Outcome:	The proposed method significantly outperforms baseline methods on three datasets.

Copied to clipboard

Xing Ma, Yangjie Zhou, Wu Sun, Zihan Liu, Jingwen Leng, Yun Lin, Shixuan Sun, Minyi Guo, Jin Song Dong

A-TIP: Attribute-aware Text Infilling via Pre-trained Language Model (2022.coling-1)

Challenge:	Existing approaches to support diverse attention variants trade performance for flexibility . expert-written kernels achieve high efficiency but are difficult to adapt .
Approach:	They propose a framework that adapts expert-written attention kernels to GPUs . they use a structured lift–transfer–lower workflow to make execution explicit .
Outcome:	The proposed framework outperforms existing frameworks and compilers on diverse variants and GPU platforms.

Copied to clipboard

Dongyuan Li, Jingyi You, Kotaro Funakoshi, Manabu Okumura

Improving Textual Network Embedding with Global Attention via Optimal Transport (P19-1)

Challenge:	Existing methods for text infilling focus on the infill length of blanks and attribute relevance, but attribute-aware content can be more useful.
Approach:	They propose an attribute-aware text infilling method via a Pre-trained language model which contains a text in filling component and a plug-and-play discriminator.
Outcome:	The proposed method improves attribute relevance without decreasing text fluency on three open-source datasets.

Copied to clipboard

Liqun Chen, Guoyin Wang, Chenyang Tao, Dinghan Shen, Pengyu Cheng, Xinyuan Zhang, Wenlin Wang, Yizhe Zhang, Lawrence Carin

Representation Learning

Challenge:	Existing methods for learning textual network embeddings are noisy and sparse.
Approach:	They propose to use text-based attention parsing to learn context-aware network embeddings.
Outcome:	The proposed model outperforms state-of-the-art methods in a number of domains.

Visual News: Benchmark and Challenges in News Image Captioning (2021.emnlp-main)

Copied to clipboard

Fuxiao Liu, Yinghan Wang, Tianlu Wang, Vicente Ordonez

SUMBT: Slot-Utterance Matching for Universal and Scalable Belief Tracking (P19-1)

Challenge:	Visual News Captioner is an entity-aware model for news image captioning . Unlike standard image captions, news images depict situations where people, locations, and events are of paramount importance.
Approach:	They propose a visual news captioner model that integrates visual and textual features to generate captions with richer information such as events and entities.
Outcome:	The proposed model can generate captions with richer information such as events and entities.

Copied to clipboard

Hwaran Lee, Jinsik Lee, Tae-Yoon Kim

Emotion Detection with Neural Personal Discrimination (D19-1)

Challenge:	Existing methods to model domain- and slot-dependent belief trackers have difficulty adding new slot-values, resulting in lack of flexibility of domain ontology configurations.
Approach:	They propose a model that captures relationships between domain-slot-types and slot-values appearing in utterances through attention mechanisms based on contextual semantic vectors.
Outcome:	The proposed model improves performance on two dialog corpora and achieves state-of-the-art accuracy.

Copied to clipboard

Xiabing Zhou, Zhongqing Wang, Shoushan Li, Guodong Zhou, Min Zhang

Multi-Granularity Interaction Network for Extractive and Abstractive Multi-Document Summarization (2020.acl-main)

Challenge:	Existing approaches to automatically predict the emotions of posts consider each post individually and predict their emotions independently.
Approach:	They propose a Neural Personal Discrimination approach to identify personal attributes from posts and connect relevant posts with similar attributes to jointly learn their emotions.
Outcome:	The proposed approach improves on existing models by capturing attributes-aware words and predicting emotions among relevant posts.

Copied to clipboard

Hanqi Jin, Tianming Wang, Xiaojun Wan

Attention is Not Only a Weight: Analyzing Transformers with Vector Norms (2020.emnlp-main)

Challenge:	Existing methods for document summarization use extractive and abstractive representations, but they don't take into account hierarchical structure of document clusters.
Approach:	They propose a multi-granularity interaction network for extractive and abstractive multi-document summarization which jointly learn semantic representations for words, sentences, and documents.
Outcome:	The proposed model outperforms baseline methods and achieves the best results on the Multi-News dataset.

Copied to clipboard

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, Kentaro Inui

TokenPenalty: Alleviating Attention Sinks and Positional Decay in LVLMs (2026.findings-acl)

Challenge:	Attention is a key component of Transformers, which have achieved considerable success in natural language processing.
Approach:	They propose to integrate attention weights and the norm of transformed input vectors into a norm-based analysis that incorporates the norm.
Outcome:	The proposed analysis shows that attention weights alone determine the output of attention and that reasonable word alignment can be extracted from attention mechanisms of Transformers.

Copied to clipboard

Xiaofeng Zhang, Yuanchao Zhu, Qiyan Zhao, Xiaosong Yuan, Jiawei Cao, Xuhang Chen

GIFT: Graph-Induced Fine-Tuning for Multi-Party Conversation Understanding (2023.acl-long)

Challenge:	Multimodal large language models (MLLMs) often hallucinate due to two relevant phenomena: massive activation phenomenon and positional information decay.
Approach:	They propose a token-level intervention strategy that dynamically suppresses irrelevant visual tokens while preserving key contextual signals.
Outcome:	Experiments show that TokenTruth significantly improves factual consistency across MLLMs on standard image understanding benchmarks.

Copied to clipboard

Jia-Chen Gu, Zhenhua Ling, Quan Liu, Cong Liu, Guoping Hu

MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformers (2026.acl-long)

Challenge:	Existing methods on understanding multi-party conversations typically embed interlocutors and utterances into sequential information flows or use superficial graph structures.
Approach:	They propose a plug-and-play method which adapts Transformer-based pre-trained language models for universal MPC understanding.
Outcome:	The proposed method can adapt Transformer-based pre-trained language models for universal MPC understanding.

Copied to clipboard

Linrui Ma, Chun Hei Lo, Xinyu Wang, Peng Lu, Xihao Yuan, Hanting Chen, Kai Han, Xinghao Chen, Chengjun Zhan, Hanlin xu, Yichun Yin, Lifeng Shang, Feng Wen, Boxing Chen, Yufei Cui

Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention (2023.findings-emnlp)

Challenge:	Existing approaches to improve efficiency often enforce rigid structural constraints such as local attention windows.
Approach:	They propose a framework that augments sparse-attention mechanisms with dynamically integrated in-context information through an efficient retrieval system.
Outcome:	Empirical results show that MATCH significantly improves the performance of sparse-attention models on synthetic and real-world natural-language tasks.

Copied to clipboard

Huiyin Xue, Nikolaos Aletras

S2GSL: Incorporating Segment to Syntactic Enhanced Graph Structure Learning for Aspect-based Sentiment Analysis (2024.acl-long)

Challenge:	Existing pre-trained language models have produced performance gains in various tasks but come with large computational requirements.
Approach:	They propose an alternative module that uses only a single shared projection matrix and multiple head embeddings (MHE) they demonstrate that MHE attention is substantially more memory efficient compared to alternative attention mechanisms.
Outcome:	The proposed model is more memory efficient compared to the current model while achieving high retention ratio on several downstream tasks.

Copied to clipboard

Bingfeng Chen, Qihan Ouyang, Yongqi Luo, Boyan Xu, Ruichu Cai, Zhifeng Hao

All You Need is Attention: Lightweight Attention-based Data Augmentation for Text Classification (2024.findings-emnlp)

Challenge:	Existing graph-based approaches to learn static structures and dynamic latent trees are lacking in incorporating semantic and syntactic information simultaneously within complex global structures.
Approach:	They propose a graph-based framework that incorporates semantic and syntactic information simultaneously within global structures.
Outcome:	The proposed framework removes irrelevant contexts and syntactic dependencies and achieves complementarity across diverse structures.

Copied to clipboard

Junehyung Kim, Sungjae Hwang

DialogConv: A Lightweight Fully Convolutional Network for Multi-view Response Selection (2022.emnlp-main)

Challenge:	Existing methods to augment text classification tasks require extensive dataset training.
Approach:	They propose a method that uses attention mechanisms to exchange semantically similar words between sentences to generate a greater diversity of synthetic sentences compared to simpler operations like random insertions.
Outcome:	The proposed method consistently outperforms baseline methods across diverse text classification conditions.

Copied to clipboard

Yongkang Liu, Shi Feng, Wei Gao, Daling Wang, Yifei Zhang

MISCA: A Joint Model for Multiple Intent Detection and Slot Filling with Intent-Slot Co-Attention (2023.findings-emnlp)

Challenge:	Existing retrieval-based dialogue systems suffer from slow inference or huge number of parameters.
Approach:	They propose a lightweight fully convolutional architecture for response selection using convolution.
Outcome:	The proposed architecture extracts matching features of context and response from 3D views.

Copied to clipboard

Thinh Pham, Chi Tran, Dat Quoc Nguyen

Knowledge-augmented Graph Neural Networks with Concept-aware Attention for Adverse Drug Event Detection (2024.lrec-main)

Challenge:	Existing models for detecting multiple intents and filling slots are based on graphs but face problems . a joint model can exploit the correlations between intents, slots and slot filling tasks .
Approach:	They propose a joint model that captures correlations between intents and slot labels . they propose MISCA to incorporate an intent-slot co-attention mechanism and a label attention mechanism .
Outcome:	The proposed model outperforms previous models on two benchmark datasets.

Copied to clipboard

Ya Gao, Shaoxiong Ji, Pekka Marttinen

Hierarchical Attention Generates Better Proofs (2025.acl-long)

Challenge:	Recent studies have used word embedding and deep learning to automate ADE detection from text, but they did not incorporate explicit medical knowledge about drugs and adverse reactions or the corresponding feature learning.
Approach:	They propose to integrate medical knowledge into ADE detection from text . they use contextualized embeddings from pretrained language models and convolutional graph neural networks to learn features differently for different types of nodes in the graph.
Outcome:	The proposed model outperforms existing models on four public datasets and shows that it is based on medical knowledge and embeddings from pretrained language models and neural networks.

Copied to clipboard

Jianlong Chen, Chao Li, Yang Yuan, Andrew C Yao

Reasoning

Challenge:	Large language models (LLMs) have shown promise in formal theorem proving, but their token-level processing often fails to capture the inherent hierarchical nature of mathematical proofs.
Approach:	They propose a regularization method that aligns LLMs’ attention mechanisms with mathematical reasoning structures and establishes a five-level hierarchy from foundational elements to high-level concepts.
Outcome:	The proposed method improves proof success rates by 2.05% on miniF2F and 1.69% on ProofNet while reducing proof complexity by 23.81% and 16.50% respectively.

From Human Reading to NLM Understanding: Evaluating the Role of Eye-Tracking Data in Encoder-Based Models (2025.acl-long)

Copied to clipboard

Luca Dini, Lucia Domenichelli, Dominique Brunato, Felice Dell’Orletta

RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarios (2026.findings-acl)

Challenge:	integrating eye-tracking features into Neural Language Models does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space.
Approach:	They used eye-gaze data from the Ghent Eye-Tracking Corpus to investigate how integrating knowledge of human reading behavior impacts Neural Language Models.
Outcome:	The proposed approach does not degrade downstream task performance, enhances alignment between model attention and human attention patterns, and compresses the embedding space.

Copied to clipboard

Yibo Zhang, Kaiwen Luo, Liang Lin, Shilinlu Yan, Jin Wang, Yaoqi Guo, Yitian Chen, Yalan Qin, Zhenhong Zhou, Kun Wang, Li Sun

MccSTN: Multi-Scale Contrast and Fine-Grained Feature Fusion Networks for Subject-driven Style Transfer (2024.lrec-main)

Challenge:	Existing evaluations rely on synthetic Gaussian noise or simplistic single-source interference, failing to capture the intricate, multi-layered acoustic dynamics that characterize authentic physical environments.
Approach:	They propose a robustness benchmark to stress-test Audio Large Models (ALLMs) using high-fidelity auditory scene simulations.
Outcome:	The proposed model performs well on a wide range of tasks, including automatic speech recognition, speech translation, and audio-based reasoning.

Copied to clipboard

Honggang Zhao, Chunling Xiao, Jiayi Yang, Guozhu Jin, Mingyong Li

Supervised Attention Mechanism for Low-quality Multimodal Data (2025.emnlp-main)

Challenge:	Stylistic style transfer is an important part of the image processing field . due to the low semantic similarity between the original image and the style image, many fine-grained style features are discarded.
Approach:	They propose a new style representation and transfer framework that can be adapted to existing image style transfers.
Outcome:	The proposed framework can be adapted to existing image style transfers.

Copied to clipboard

Sijie Mai, Shiqin Han, Haifeng Hu

Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers (2025.emnlp-main)

Challenge:	Current studies address missing and noisy modalities separately in multimodal data . missing modality is often caused by unavailable data collection equipment or sensor failures .
Approach:	They propose a framework for multimodal affective computing that addresses missing and noisy modalities to enhance model robustness in low-quality data scenarios.
Outcome:	The proposed model outperforms state-of-the-art baselines on multiple datasets under the settings of complete modalities, missing modalités, and noisy modality.

Copied to clipboard

Yukun Zhang, Xueqing Zhou

Attention as Selector: Unlocking VLM Attention for Long Document Page Retrieval (2026.acl-long)

Challenge:	Existing approaches to optimize attention for long sequences have been limited by their computational cost.
Approach:	They propose a framework that infuses partial differential equations into the Transformer’s attention mechanism to better handle long sequences.
Outcome:	The proposed framework achieves consistent performance gains over standard and long-sequence Transformer variants across a range of tasks.

Copied to clipboard

Minfeng Zhu, Linxin Bao, Wei Chen, Linchao Zhu

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025.acl-long)

Challenge:	Existing page-level retrieval methods lack query–page interaction before similarity scoring . Existing methods require large-scale datasets to align visual and textual embeddings .
Approach:	They propose a retrieval framework that utilizes attention mechanisms inside VLMs for page selection.
Outcome:	The proposed retrieval framework outperforms embedding-based retrieval methods on four long-document benchmarks.

Copied to clipboard

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Yuxing Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, Wangding Zeng

Mamba Knockout for Unraveling Factual Information Flow (2025.acl-long)

Challenge:	Long-context modeling is crucial for next-generation language models, but high computational cost of standard attention mechanisms poses significant computational challenges.
Approach:	They propose a natively trained Sparse Attention mechanism that integrates algorithms with hardware-aligned optimizations to achieve efficient long-context modeling.
Outcome:	The proposed model maintains or exceeds Full Attention models across general benchmarks, long-context tasks, and instruction-based reasoning.

Copied to clipboard

Nir Endy, Idan Daniel Grosbard, Yuval Ran-Milo, Yonatan Slutzky, Itay Tshuva, Raja Giryes

Latent-Condensed Transformer for Efficient Long Context Modeling (2026.acl-long)

Challenge:	Recent work has introduced Mamba-based SSM architectures that rival Transformer performance in various settings.
Approach:	They propose to use attentional interpretability techniques originally developed for Transformers to trace how information is transmitted and localized across tokens and layers.
Outcome:	The proposed model disentangles how distinct features enable token-to-token information exchange or enrich individual tokens, thus offering a unified lens to understand Mamba internal operations.

Copied to clipboard

Zeng You, Yaofo Chen, Qiuwu Chen, Ying Sun, Shuhai Zhang, Yingjian Li, Yaowei Wang, Mingkui Tan

From Logical to Computational Sparsity: Structure-Aware Block-Sparse Attention for Long-Code Completion (2026.acl-long)

Challenge:	Existing approaches address these bottlenecks separately: Multi-head Latent Attention (MLA) reduces the KV cache by projecting tokens into a low-dimensional latent space, while sparse attention reduces computation.
Approach:	They propose a Latent-Condensed Attention mechanism that performs structured context condensation directly within MLA's latent space.
Outcome:	The proposed approach reduces KV cache size and attention cost without adding parameters.

Copied to clipboard

Yanli Wang, Yanlin Wang, Bowen Zhang, Yiwei Zhang, Daya Guo, Jiachi Chen, Hongyu Zhang, Zibin Zheng

Programming Languages in NLP

Challenge:	Existing sparse attention methods for long-context generation pose high latency . general sparsity methods cause excessive accuracy degradation without considering code structure .
Approach:	They propose a training-free Structure-aware block-sparse attention mechanism that bridges the gap between logical and computational sparsity.
Outcome:	The proposed method reduces TTFT by 45-55% while maintaining accuracy within 3% of dense attention.

RRAtention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference (2026.acl-long)

Copied to clipboard

Siran Liu, Guoxia Wang, Sa Wang, Jinle Zeng, Haoyang Xie, Siyu Lou, Jiabin Yang, Dianhai Yu, Haifeng Wang, Chao Yang

Visual Data in NLP

Challenge:	Existing approaches to dynamic sparse attention require preprocessing, lack global evaluation, violate query independence, or incur high computational overhead.
Approach:	They propose a dynamic sparse attention method that achieves all desirable properties through a head round-robin (RR) sampling strategy.
Outcome:	Experiments on natural language understanding and multimodal video comprehension show that the proposed method achieves 2.4 speedup at 128K context length outperforming existing methods.

CSI: An Investigative Multi-Agent Framework for Explainable Short Video Fake News Detection (2026.findings-acl)

Copied to clipboard

Yuxin Wang, Yang Yang, Huaiwen Zhang

Refining Attention for Explainable and Noise-Robust Fact-Checking with Transformers (2025.emnlp-main)

Challenge:	Existing methods for short video fake news detection rely on black-box MSLMs with poor explainability and superficial understanding or on specific prompt strategies for Multimodal Large Language Models (MLLMs)
Approach:	They propose a multi-agent framework called CSI for short video fake news detection.
Outcome:	The proposed framework provides rigorous explanations while achieving state-of-the-art performance on two real-world datasets.

Copied to clipboard

Jean-Flavien Bussotti, Paolo Papotti

QuASAR: A Question-Driven Structure-Aware Approach for Table-to-Text Generation (2025.acl-long)

Challenge:	Conventional transformer-based models falter due to noise sensitivity and lack explainability . ATTUN is a transformer architecture designed to enhance model transparency and resilience to noise.
Approach:	They propose a transformer architecture that enhances model transparency and resilience to noise . ATTUN is a module that directly modifies attention weights . they validated their approach using fact-checking datasets based on their results .
Outcome:	The proposed model improves predictions and identify relevant sections of input data.

Copied to clipboard

WeiJie Liu, Yibin Zheng, Fang Kong

Text Generation

Challenge:	Existing methods for table-to-text generation fail to capture the structure of tabular data or rely on complex attention mechanisms, limiting their applicability.
Approach:	They propose a question-driven self-supervised approach to enhance the model’s structural perception and representation capabilities by focusing on structure-related queries.
Outcome:	The proposed model improves its model's structural perception and representation capabilities by guiding it to capture local and global table structures.

Mitigating Structural Knowledge Collapse in Domain-Specific LLMs via Morpheme-Aware KV-Aggregation (2026.acl-long)

Copied to clipboard

Yuxuan Si, Zheqi Lv, Chengxi Zang, Zhengyu Chen, Fei Wu

Representation Learning

Challenge:	Existing tokenizers over-fragment domain terms, disrupting morpheme semantics.
Approach:	They propose a lightweight tokenizer that dynamically consolidates fragments without tokenizer changes.
Outcome:	The proposed adapter outperforms vocabulary adaptation baselines on medical and legal terms by 3.2–4.6% and 7.9% on high-fragmentation terms.

Probing Relative Interaction and Dynamic Calibration in Multi-modal Entity Alignment (2025.acl-long)

Copied to clipboard

Chenxiao Li, Jingwei Cheng, Qiang Tong, Fu Zhang, Cairui Wang

Can Language Models Follow Multiple Turns of Entangled Instructions? (2025.findings-emnlp)

Challenge:	Current methods for multi-modal entity alignment ignore relative interactions between modalities and the accuracy of weights.
Approach:	They propose a relative interaction and calibration framework for multi-modal entity alignment that uses attention mechanisms to perceive the uncertainty of the weight for each modality.
Outcome:	The proposed framework outperforms baselines across 5 datasets and 23 settings.

Copied to clipboard

Chi Han, Xin Liu, Haodong Wang, Shiyang Li, Jingfeng Yang, Haoming Jiang, Zhengyang Wang, Qingyu Yin, Liang Qiu, Changlong Yu, Yifan Gao, Zheng Li, Bing Yin, Jingbo Shang, Heng Ji

HyperAdaLoRA: Accelerating LoRA Rank Allocation During Training via Hypernetworks without Sacrificing Performance (2026.findings-acl)

Challenge:	Despite of significant achievements in improving instruction-following capabilities of large language models, the ability to process multiple potentially entangled or conflicting instructions remains a considerable challenge.
Approach:	They construct multi-turn instruction with 1.1K high-quality multi-turned conversations using the human-in-the-loop approach and examine their capabilities.
Outcome:	The proposed model shows that it is difficult to integrate multiple turns and balance competing objectives when instructions intersect or conflict.

Copied to clipboard

Hao Zhang, Zhenjia Li, Yifan Gao, Xi Xiao, Heng Zhang, Shuyang Zhang, null Xiaoxincc, Bo Huang, Yuhang Wu, Tianyang Wang, Hao Xu

Vocabulary Hijacking in LVLMs: Unveiling Critical Attention Heads by Excluding Inert Tokens to Mitigate Hallucination (2026.acl-long)

Challenge:	Low-Rank Adaptation (LoRA) assumes a uniform rank r for each incremental matrix, not accounting for the varying significance of weight matrices across modules and layers.
Approach:	They propose a framework that allows for faster convergence of low-rank adaptive models . they use a hypernetwork to prune the outputs of the hypernetworks to generate parameters .
Outcome:	The proposed framework accelerates convergence of AdaLoRA by leveraging a hypernetwork.

Copied to clipboard

Yangneng Chen, Junlin Li, Weijun Yao, Xilai Ma, Guodong DU, Wenya Wang, Jing Li